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ABSTRACT 

Research can be related to reality, to meaning, and 
to decisions and decision makers. There are some assumptions about 
reality and research that are taken for granted: a) reality does 
indeed exist--which one ma3t accept as true; b) research is an 
accurate portrait of reality — although no research has ever done 
justice to the original object; c) the methodology used, has little 
effect on the outcome of the study — although, in truth, the 
methodology partly determines the answer; d) prediction is 
possible — although experience shows that things do change and that 
there are intervening accidents of history. The problem is an 
unawareness of the biases of assumptions. Similarly with regard tq 
meaning and reality , the positive mania with statistical significance 
must be ended and the premise that statistical significance may have 
little relationship to practical significance must be acknowledged. 
Consequently, one tends to be a "closed shop" regarding his own 
expertise. In overall summary, although reality is assumed to exist, 
the way of finding images of that reality is often inaccurate, narrow 
and, all too often, irrelevant and confusing. (JA) 
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This paper discusses the relationship of recoarch to reality, research 
to meaning, and research to decisions and decision makers. The two 
primary foci are on the assumptions we take for granted but which 
color our research and our findings, and on the difficulties decision makers 
have in using our findings. 

Reality. The first assumption is interesting to deal with but difficult 
to resolve. It is that reality does indeed exist. Without getting into the 
problem presented by the Taoist teactier who fell asleep under a bow tree 
and dreamed he was a butterfly, and when he awoke he could not determine 
whether he was then a man dreaming that he was a butterfly, or now a 
butterfly dreaming he was a man, let us for the sake of mental and 
professional health, assume with most of Western society that indeed 
this building does exist and that the problems faced by decision makers 
do exist. 




Second, after we assume that reality exists, we make another 



assumption which directly affects the reliability and validity of our 



portrayal of reality. When we have said that we have described a 



research statements. We assume that our research is an accurate 



phenomenon or that two variables are significantly different, what 



have we actually said? Have we truly described the phenomenon or 
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have we given an image of the phenomenon? One is reminded of Plato's 
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allegory of the cavo where the actual object is never seen by the person, 
but only the shadow of the object as cast by a fire behind it. When we do 
research, we arc in much the same situation in that what we describe is 
greater than ovir description. No research of which wc are aware has 
ever done justice to the original object. We merely cast images on a 
wall. If this is true, then all our research statements must, by definition, 
be false to the extent tliat they are not completely true. This problem 
becomes even greater when we research abstract problems such as learning, 
power, value, growth, or our own field of juvenile delinquency. We 
should keep in mind that our descriptions of reality are merely approxi- 
mations and that these approximations are heavily weighted by our 
methodologies. 

Third, regarding methodology, our assumption is that the methodology 
usod has little effect on the outcome of the study. It is the position of this 
paper that the methodology partly determines the answer. The empirical 
method as most commonly used leads to quantified and differentiated 
answers. If an engineer were to determine whether a dam should be 
built on a river, the answer would be in quantifiable terms such as 
terrainian conditions ana water volume. If a poet were asked the same 
question, his answer would be in qualitative terms such as beauty, 
harmony, composition, etc. The realities presented would be quite 
different. In educational reseaoich our penchant has been to quantify 
rather than to qualify, and our realities reflect this bias. It has been 
the practice of educational research that if you can't count it, it isn't 



worth counting and wc rarely tako into account the effect this has 
on outcomes. 

A fourth assumption of educational research is that prediction is 
possible. * We further assume that we can predict a particular variable not 
in terms of its greater context bvit rather as if the variable existed in 
a vacuum. We assume t hat all those artifacts which are not inclvided 
in our study either have no effect on the studied variables or that they will 
not change or that any changes v/hich occur will be random and slight and 
not affect the outcome. And yet, our own experience tells us that things 
do indeed change and that our predictions are at best inaccurate because 
of the intervening accidents of history. For example, predictions about 
everything from learning to juvenile delinquency were affected by the 
automobile, or the pill, or television which have no direct relationship 
to those predictions but which fall under the category of accidents or 
intervening variables which could not be accounted for in our predictions. 
What then are we saying when we attempt to predict? We are saying, 
"All things being equal, using my quantified approach and assuming that 
history will not intervene, I think the following things will happen again 
as they did before. " The primary problem is that all things are not equal 
and, as Heraclitus said, "You can't put your foot in the same stream twice. 
This puts us in the difficult position of either dealing with such narrow 
constructs that history canU intervene, or making predictive statements 
that, at bes.t, are misleading, or alternately, including broader tested 
methodologies in our research. 



In sumni^iry, the pvoblcm is not that \vc mako assumptions, but 
rather, that wc often arc not aware of thomnor of the biases which 
they introduce into our research. 

Meanings The problem of the meaning of research is related to, but 
separate from, that of the reality of research. More and more the 
audience for which we perform research and evaluation is made up of 
non-university institutions. In the past, if we made assumptions which 
biased our data, few people really noticed since our audience was made 
up of our colleagues who functioned under the same assumptions and 
biases. However, if research is to become more than an interesting 
intellectual exercise and begin dealing with the audience of decision 
makers, then a number of changes have to be made. 

First, we must end our positive mania with statistical significance. 
Not finding significance may be just as important to decision makers as 
findiry it. Determining that we can not find relationships between 
variables is just as interesting as finding that there is a relationship. 
Our published findings are also becoming suspect. There is a decided 
penchant for editors to publish articles which have statistical significance 
above those which do not. There are some journals which state that if 
statistical significance is not reached they will not consider the article^ 
and there are many researchers who will not submit articles which do not 
reach significance. If a journal rejects nineteen out of twenty articles 
and uses statistical significance as a major criterion, it is entirely 
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possible tliat the article which is published is the one out of twenty 
which is a Type J. error. We arc by no means suggesting that this is 
the case. What we are suggesting is that editors look caref uliy at 
the research implications of editorial policy since their policies 
may be misleading decision makers and researchers alike. Leaving 
aside the problem of trying to replicate only statistically significant 
studies, our present problem is that while decision makers are 
interested in making decisions, they are just as interested in not 
maki/ig incorrect decisions. Therefore, our mania with statisitcal 
significance is no more interesting to them than a finding of non- 
significance. 

Second, finding statistical significance may have little relationship 
to finding practical significance. If the number of subjects in the study 
is sufficiently large, then almost any statistical difference or correlation 
can be statistically significant. It should always be remembered, however, 
that the decision maker deals with practicality and his decisions are 
based on such mundane criteria as the number of votes gained, the 
amount of money saved, or the efficiency increased. Statistical significance 
may or may not give him answers to those questions. 

Third, because our audience has been made up of our colleagues, we 
have developed a series of statistical procedures which are highly refined. 
These procedures have three basic drawbacks for use in the field: 1. they 
are far too refined for the rather crude data found in the field; 2. many 
institutions are no longer willing to commis^on new studies to meet 
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tho proconcoivcd designs and methodologies cf the researcher when most 
of the requisite data have already been gathere d for administrative 
purposes and merely need to be analyzed; 3. the answers which are generated 
by refined statistical procedures are usually unintcrpr etable to the decision 
maker, A statistically significant **F test*^ or the factor loadings from a 
factor analysis are so much Greek to the decision maker. He is put in 
the position of either dismissing the data analysis as so much intellectual 
jargon, accepting the researcher's interpretation on faith, or bypassing 
the university researcher in favor of his own evaluation staff which is 
taught to discard everything they learned at the university during their 
first few weeks at the job. 

Fourth, to our own detriment and that of the community we have 
maintained a closed shop regarding our expertise. This point is- related 
to the third in that our procedures are so complicated both in their 
execution and their interpretation that it is difficult if not impossible for 
the decision maker to learn them. Further, however, we have made little 
effort to go into the field and teach our procedures and to learn procedures 
which may be of help to the decision makers and to ourselves. 

Fifth, there is a feeling of '^purity" among many researchers. The 
feeling is that if we go into the field and deal with decision problems, we 
can no longer consider ourselves pure or basic researchers. It is the 
opinion of these authors that universities will no longer be supported by a 
comrrunity which does not see some direct, relationship between solving 
^ their problems and one of the most expensive institutions in their midst. 
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We should recall that our statistical rncthoc have at thoir root the 
determination of nnethods to win at games of chance an "impure" 
and highly practical probl:;m. 

Finally, in overall summary, we choose to assume, with most of 
our colleagues, that reality does indeed exist. However, the way in 
which we go about finding images of that reality is often inaccurate and 
narrow and all too often is irrelevant and confusing. Our research has 
little relevance to decision making because it is too complex, it is 
largely uninterpr etahle , and our methods are inappropriate to the 
data or to the audience. If we wish to continue research as an interesting 
intellectual exercise for the amusement and edification of our colleagues, 
we can continue in much the same fashion. - If, however, we wish to 
aid decision makers we must mend our ways and take as our base 
the problem as presented, not our preconceived notions and assumptions 
of classic re sear ch de sign. 



